HyNNA: Improved Performance for Neuromorphic Vision Sensor based Surveillance using Hybrid Neural Network Architecture

Applications in the Internet of Video Things (IoVT) domain have very tight constraints with respect to power and area. While neuromorphic vision sensors (NVS) may offer advantages over traditional imagers in this domain, the existing NVS systems either do not meet the power constraints or have not demonstrated end-to-end system performance. To address this, we improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal and address the low-power requirement for object detection and classification by exploring various convolutional neural network (CNN) architectures. Specifically, we compare the results obtained from our object detection framework against the state-of-the-art low-power NVS surveillance system and show an improved accuracy of 82.16% from 63.1%. Moreover, we show that using multiple bits does not improve accuracy, and thus, system designers can save power and area by using only single bit event polarity information. In addition, we explore the CNN architecture space for object classification and show useful insights to trade-off accuracy for lower power using lesser memory and arithmetic operations.Comment: 4 pages, 2 figure

Low-power dynamic object detection and classification with freely moving event cameras

We present the first purely event-based, energy-efficient approach for dynamic object detection and categorization with a freely moving event camera. Compared to traditional cameras, event-based object recognition systems are considerably behind in terms of accuracy and algorithmic maturity. To this end, this paper presents an event-based feature extraction method devised by accumulating local activity across the image frame and then applying principal component analysis (PCA) to the normalized neighborhood region. Subsequently, we propose a backtracking-free k-d tree mechanism for efficient feature matching by taking advantage of the low-dimensionality of the feature representation. Additionally, the proposed k-d tree mechanism allows for feature selection to obtain a lower-dimensional object representation when hardware resources are limited to implement PCA. Consequently, the proposed system can be realized on a field-programmable gate array (FPGA) device leading to high performance over resource ratio. The proposed system is tested on real-world event-based datasets for object categorization, showing superior classification performance compared to state-of-the-art algorithms. Additionally, we verified the real-time FPGA performance of the proposed object detection method, trained with limited data as opposed to deep learning methods, under a closed-loop aerial vehicle flight mode. We also compare the proposed object categorization framework to pre-trained convolutional neural networks using transfer learning and highlight the drawbacks of using frame-based sensors under dynamic camera motion. Finally, we provide critical insights about the feature extraction method and the classification parameters on the system performance, which aids in understanding the framework to suit various low-power (less than a few watts) application scenarios

e-TLD: Event-based Framework for Dynamic Object Tracking

This paper presents a long-term object tracking framework with a moving event camera under general tracking conditions. A first of its kind for these revolutionary cameras, the tracking framework uses a discriminative representation for the object with online learning, and detects and re-tracks the object when it comes back into the field-of-view. One of the key novelties is the use of an event-based local sliding window technique that tracks reliably in scenes with cluttered and textured background. In addition, Bayesian bootstrapping is used to assist real-time processing and boost the discriminative power of the object representation. On the other hand, when the object re-enters the field-of-view of the camera, a data-driven, global sliding window detector locates the object for subsequent tracking. Extensive experiments demonstrate the ability of the proposed framework to track and detect arbitrary objects of various shapes and sizes, including dynamic objects such as a human. This is a significant improvement compared to earlier works that simply track objects as long as they are visible under simpler background settings. Using the ground truth locations for five different objects under three motion settings, namely translation, rotation and 6-DOF, quantitative measurement is reported for the event-based tracking framework with critical insights on various performance issues. Finally, real-time implementation in C++ highlights tracking ability under scale, rotation, view-point and occlusion scenarios in a lab setting.Comment: 11 pages, 10 figure

EBBIOT: A Low-complexity Tracking Algorithm for Surveillance in IoVT Using Stationary Neuromorphic Vision Sensors

In this paper, we present EBBIOT-a novel paradigm for object tracking using stationary neuromorphic vision sensors in low-power sensor nodes for the Internet of Video Things (IoVT). Different from fully event based tracking or fully frame based approaches, we propose a mixed approach where we create event-based binary images (EBBI) that can use memory efficient noise filtering algorithms. We exploit the motion triggering aspect of neuromorphic sensors to generate region proposals based on event density counts with >1000X less memory and computes compared to frame based approaches. We also propose a simple overlap based tracker (OT) with prediction based handling of occlusion. Our overall approach requires 7X less memory and 3X less computations than conventional noise filtering and event based mean shift (EBMS) tracking. Finally, we show that our approach results in significantly higher precision and recall compared to EBMS approach as well as Kalman Filter tracker when evaluated over 1.1 hours of traffic recordings at two different locations.Comment: 6 pages, 5 figure

EBBINNOT: A Hardware Efficient Hybrid Event-Frame Tracker for Stationary Dynamic Vision Sensors

As an alternative sensing paradigm, dynamic vision sensors (DVS) have been recently explored to tackle scenarios where conventional sensors result in high data rate and processing time. This paper presents a hybrid event-frame approach for detecting and tracking objects recorded by a stationary neuromorphic sensor, thereby exploiting the sparse DVS output in a low-power setting for traffic monitoring. Specifically, we propose a hardware efficient processing pipeline that optimizes memory and computational needs that enable long-term battery powered usage for IoT applications. To exploit the background removal property of a static DVS, we propose an event-based binary image creation that signals presence or absence of events in a frame duration. This reduces memory requirement and enables usage of simple algorithms like median filtering and connected component labeling for denoise and region proposal respectively. To overcome the fragmentation issue, a YOLO inspired neural network based detector and classifier to merge fragmented region proposals has been proposed. Finally, a new overlap based tracker was implemented, exploiting overlap between detections and tracks is proposed with heuristics to overcome occlusion. The proposed pipeline is evaluated with more than 5 hours of traffic recording spanning three different locations on two different neuromorphic sensors (DVS and CeleX) and demonstrate similar performance. Compared to existing event-based feature trackers, our method provides similar accuracy while needing approx 6 times less computes. To the best of our knowledge, this is the first time a stationary DVS based traffic monitoring solution is extensively compared to simultaneously recorded RGB frame-based methods while showing tremendous promise by outperforming state-of-the-art deep learning solutions.Comment: 16 pages, 13 figure

Laparoscopic cholecystectomy and common bile duct exploration using choledochotomy and primary closure following failed endoscopic retrograde cholangiopancreatography: A multicentric comparative study using three-port vs multiport

Background: Laparoscopic surgery has changed many ways in which we as surgeons manage patients, offering better results, quicker recovery, and fewer complications using minimally invasive techniques, especially in common bile duct (CBD) surgery. Not only can laparoscopic techniques be applied to programed surgery but also emergencies and those following failed endoscopic retrograde cholangiopancreatography (ERCP). Objectives and aims: Describe and compare clinical and surgical results of the laparoscopic CBD exploration with primary closure using a 3-port vs multiport approach. Materials and methods: We present a multicentric comparative study of 197 consecutive patients who underwent a laparoscopic gallbladder removal along with CBD exploration with primary closure following failed (ERCP to extract CBD stones; 104 patients were managed by three-port vs 93 multiport laparoscopic surgery in five centers of Bogotá, Colombia, between 2013 and 2017 with follow-up of 1 year. Results: A total of 197 patients were taken to laparoscopic gallbladder removal along with CBD exploration with primary closure, 104 patients via three-port technique and 93 patients via multiport. All (100%) the patients had previously failed ERCP. The average surgical time on the three-port approach was 106 minutes vs 123 minutes on multiport. Only in the multiport technique we had an average conversion of 2%. Mean hospital stay of 2.5 days, less for the three-port approach vs multiport in 5–7 days. There was a need of reintervention in 1% of the patients who underwent three-port exploration. Conclusion: Postoperative pain, use of an additional port, complication rates, operation time, and cost of the three-port technique were similar to those of the conventional approach. Large randomized controlled trials are needed to examine the true benefits of the three-port technique

Laparoscopic choledochal cyst resection with simplified common bile duct reconstruction in an adult population: A case series

Introducción: Los quistes de colédoco (CC) son una patología congénita poco frecuente en la población adulta. Desde 1995, el manejo laparoscópico ha sido descrito para esta entidad. Sin embargo, su manejo se considera un tema controvertido debido al aumento del riesgo de colangiocarcinoma. Materiales y métodos: se realizó un estudio retrospectivo, observacional y descriptivo considerando pacientes diagnosticados de CC que fueron operados en un centro de referencia de cirugía hepatobiliar desde enero de 2013 hasta junio de 2018. Los pacientes fueron llevados a laparoscopia simplificada hepaticoyeyunostomía con reconstrucción en Y de Roux. Se presenta un análisis retrospectivo de los datos obtenidos. Resultados: Diez pacientes adultos con CC fueron sometidos a reconstrucción biliar quirúrgica a una edad media de 34,5 años; 75% tenía CC Todani tipo I y 25% Todani tipo IV-B CC. Aproximadamente el 50% de los pacientes fueron diagnosticados mediante colangiopancreatografía retrógrada endoscópica (CPRE) y el 50% de ellos mediante colangiopancreatografía por resonancia magnética. Ninguno requirió reintervención, no se informó mortalidad; y la estancia media en el hospital fue 5 días, ningún paciente tuvo fuga biliar posoperatoria, ninguno se convirtió a cirugía abierta y todos los pacientes tuvieron una tolerancia adecuada a la alimentación oral 2 días postoperatorios. El seguimiento a largo plazo no mostró incidencia de colangiocarcinoma después de un seguimiento de 2 años. Conclusión: Los quistes de colédoco en adultos son una patología poco frecuente que tiene una alta probabilidad de desarrollar malignidad cuando no se realiza de forma adecuada quirúrgicamente manejado y debido al reflujo biliar secundario. Estos factores hacen que el manejo quirúrgico sea una decisión crítica. El abordaje laparoscópico simplificado presentado en este trabajo parece ser una alternativa eficaz y segura a la cirugía reconstructiva de la vía biliar.Introduction: Choledochal cysts (CC) are rare congenital pathology in adult population. Since 1995, laparoscopic management has been described for this entity. Nevertheless, its management is considered to be a controversial matter due to the augmented risk of associated cholangiocarcinoma. Materials and methods: A retrospective, observational, and descriptive study was conducted considering patients diagnosed with CC who were operated at a hepatobiliary surgery referral center from January 2013 to June 2018. Patients were taken to simplified laparoscopic hepaticojejunostomy with a Roux-en-Y reconstruction. A retrospective analysis of the data obtained is presented. Results: Ten adult patients with CC underwent surgical biliary reconstruction at a mean age of 34.5 years; 75% had Todani type I CC and 25% Todani type IV-B CC. About 50% of the patients were diagnosed via endoscopic retrograde holangiopancreatography (ERCP) and 50% of them via magnetic resonance cholangiopancreatography. None required re-intervention, no mortality was reported; and the mean hospital stay was 5 days, no patient had postoperative biliary leakage, none was converted to open surgery, and all patients had adequate oral feeding tolerance 2 days postoperative. Long-term follow-up showed no incidence of cholangiocarcinoma after 2-year follow-up. Conclusion: Choledochal cysts in adults is a rare pathology that has a high probability of developing malignancy when not adequately surgically managed and because of secondary bile reflux. These factors make surgical management a critical decision. The simplified laparoscopic approach presented in this paper seems to be an effective and safe alternative to biliary duct reconstructive surgery

HyNNA : improved performance for neuromorphic vision sensor based surveillance using hybrid neural network architecture

Applications in the Internet of Video Things (IoVT) domain have very tight constraints with respect to power and area. While neuromorphic vision sensors (NVS) may offer advantages over traditional imagers in this domain, the existing NVS systems either do not meet the power constraints or have not demonstrated end-to-end system performance. To address this, we improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal and address the low-power requirement for object detection and classification by exploring various convolutional neural network (CNN) architectures. Specifically, we compare the results obtained from our object detection framework against the state-of-the-art low-power NVS surveillance system and show an improved accuracy of 82.16% from 63.1%. Moreover, we show that using multiple bits does not improve accuracy, and thus, system designers can save power and area by using only single bit event polarity information. In addition, we explore the CNN architecture space for object classification and show useful insights to trade-off accuracy for lower power using lesser memory and arithmetic operations